Apparatus for audio dictation and navigation of electronic images and documents

ABSTRACT

Apparatus for annotation and navigation of electronic images comprising a handheld controller for use with a computer system. The controller comprises a microphone, voice dictation controls that generate commands to operate a digital audio recording and playback functions in the computer system, and image display controls to advance or back up through a series of images or files displayed in an image display unit of said computer system.

[0001] This application is based on, and I claim priority rights under35 U.S.C. §119(e) from, U.S. Provisional patent application ser. No.60/021,359, filed Jul. 8, 1996.

FIELD OF THE INVENTION

[0002] The present invention relates generally to the field ofcomputerized image acquisition, management and display, with particularuse in the field of medical applications thereof.

BACKGROUND OF THE INVENTION

[0003] Radiographic images, such as X-ray films, and Magnetic resonanceimaging (“MRI”) and computerized tomograph (CT) transparencies havetraditionally been reviewed by a diagnosing physician such as aradiologist on a film viewer, on which a series of images in a patientstudy are displayed in groups of two or more. Different image series,generated by processing the data captured during a patient scan indifferent ways, are typically provided. The different image series maydisplay the same portions of the patient's anatomy but processingreduces the amount of information in the image to allow for films havingsufficient pixel depth to provide the detail needed for diagnosis in thetissue of interest. The film viewer may be a static light box or amotorized film viewer. A motorized film viewer consists of verticallypositioned light boxes and an endless belt on which the films aremounted and which passes the films in front of the light boxes.Motorized film viewers are controlled by the radiologist using footpedals which activate the motorized viewer to cause the films to bemoved to the left or to the right (or in some cases, up and down) topermit viewing of the desired portions of the film series. The imagesare usually presented in a standard sequence, usually in the order ofimage capture. The radiologist will typically dictate a medical studyinto a tape recorder while viewing the films, for later transcription.The films and transcribed dictation are stored in a patient's file.

[0004] Filmless image systems have also been proposed and used. Filmlesssystems have been used in connection with handling both MRI images, andcomputer tomography images generated by processing X-ray image data togenerate a series of “slices” through the human body. Such systemshistorically have presented all the image data captured by the MRI ortomography equipment, as opposed to in film images (where the medicaltechnician will expose films only for selected series of images ofclinical interest). This mass of information has proven unwieldy toreview and consider, as instead of filtering the image data through theselection of a skilled technician, all the image data is presented tothe diagnosing physician. Another disadvantage of this approach is thatit does not provide an “image of record,” that is, the basic image(s)relied on by the physician in making the diagnosis. Given thelimitations of the image processing equipment, it took a significantamount of time to review the numerous image series. Such systems are farless efficient than the conventional light boxes, which allowed theradiologist to rapidly review numerous series of images to find the fewimages showing the pathological condition of interest. Other filmlesssystems have been proposed in which a series of images are presented tothe diagnosing physician simultaneously. One such system is described inHilton, U.S. Pat. No. 5,452,416.

[0005] However, all of these systems have proven awkward in actual use,as they require use of mouse, keyboard, trackball or other device tonavigate through the image series. These control devices severely impairthe diagnosing physician's ability to concentrate on the imagespresented and to relate the images to his knowledge built up throughyears of practicing medicine. The constant need to correctly positionand actuate a mouse and trackball to navigate through images has madethe use of such systems far less efficient than conventional filmsystems. In conventional film systems, the image navigation iseffortless, requiring no more than movement of the eyes over a wallfilled with film images, allowing the diagnosing physician toconcentrate on the real work of interpreting the medical images to reacha diagnosis. Such control devices thus limit the fluid navigation by thediagnosing physician through the image series, and tie the physician toa single position. These control devices are also awkward for adiagnosing physician who may need to refer to materials such astextbooks or prior film-based medical studies for comparison. Thesedevices also may impede the physician's ability to easily dictate themedical study, as most dictation equipment requires a handheld recorderor microphone, thus requiring the physician to either put down and pickup the microphone each time he needs to activate the mouse or trackballto see further images. Alternatively, the physician might learn to worktwo handedly, but this creates additional complications, for example, ifthe physician also desired to turn pages in or otherwise refer to a textor other reference material.

SUMMARY OF THE INVENTION

[0006] It is an object of the invention to provide a apparatus andmethod for annotation and navigation of electronic images and documentsthat increases the diagnosing physician's efficiency in using thesystem.

[0007] It is an object of the invention to provide such an apparatusthat can be operated single handedly. It is an object of the inventionto provide such a system which permits the diagnosing physician to movearound in a work area to retrieve reference works, view past studies, ordo other things while at the same time continuing with the medical studyof the images.

[0008] These objects, and other objects as disclosed in thisapplication, are achieved by an apparatus for audio dictation andnavigation of electronic images for use with a personal computer system.The computer system is provided with at least one image display unit,random access memory, means for digital data storage, and an analog todigital conversion system. The apparatus comprises a handheld wirelesscontroller having a combination of controls serving as an image controlactuator and a dictation actuator. The image control actuator isoperable to transmit wireless control signals to a wireless controlsignals receiving unit operably connected to the computer system tocause the computer system to display each one of a series of electronicimages, in sequential order, and to select display of each next image ornext preceding image, and to cause display of related image series insequential order, and to select display of each next image series ornext preceding image series. The dictation actuator is operable toactivate wireless transmission of voice signals, generated by amicrophone, to a wireless voice signals receiving unit operablyconnected to the computer system to cause the computer system to convertthe voice signals to digital signals and to record the digitized voicesignals as digital audio files in the means for digital data storage,and to permit selective playback of such files. Further details of theinvention are described below.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The foregoing aspects and other features of the invention areexplained in the following description, taken in connection with theaccompanying drawings, wherein:

[0010]FIG. 1 is a top rear perspective view of a handheld apparatus foraudio dictation and navigation of electronic images and documents inaccordance with one embodiment of the invention.

[0011]FIG. 2 is a top rear perspective view of a handheld apparatus foraudio dictation and navigation of electronic images and documents inaccordance with another embodiment of the invention.

[0012]FIG. 3 is a bottom front perspective view of the handheldapparatus for audio dictation and navigation of electronic images anddocuments of FIG. 2.

[0013]FIG. 4 is a front schematic illustration of computers of aninterpretation unit for use with the handheld apparatus of FIGS. 1, 2and 3.

[0014]FIG. 5 is a front elevation view of an embodiment of aninterpretation unit.

[0015]FIG. 6 is a top plan view of the monitors of an interpretationunit positioned for use by a single person.

[0016]FIG. 7 is a top plan view of the monitors of an interpretationunit positioned for use by multiple persons.

DETAILED DESCRIPTION OF THE INVENTION

[0017] Referring to FIGS. 1-7, where like numbered elements in thedrawings represent the same elements, a handheld controller 20 for audiodictation and navigation of electronic images and documents is shown inFIG. 1, and another embodiment of such a controller is shown in FIGS. 2and 3 as 120.

[0018] Controllers 20, 120 comprise two significant control subsystems.An image control subsystem permits navigation among the images to bedisplayed by a viewing unit. A dictation subsystem operates to receivevoice dictation. Preferably, a wireless transmitter subsystem transmitscommands from the image control subsystem and the dictation controlsubsystem to a personal computer based interpretation unit 22.

[0019] Referring now to FIGS. 4 and 5, the interpretation unit 22 ispreferably provided with at least one conventional personal computer orworkstation system 80. Most preferably, interpretation unit 22 comprisesthree conventional personal computer systems 80 which are linkedtogether by a network system. Each personal computer system 80 ispreferably a conventional personal computer such as an Intel processorbased system running a Windows 95 or Windows NT operating system. Eachpersonal computer system 80 includes an image display unit 82 such as a20 inch by 20 inch 0.28 dpi non interlaced monitor, random access memory(preferably a minimum of 32 megabytes, and more preferably 128megabytes), means for digital data storage such as a hard disk drive,and an analog to digital conversion system for converting analog audiosignals to digital format. The use of three linked personal computersprovides a simple and effective way of driving three separate monitors,which can be independently controlled by the handheld controllers 20,120 or which can be controlled by the handheld controllers 20, 120 via aselected “master” personal computer 84, with the flanking left personalcomputers 86 and right personal computer 88 acting as “slave” computersunder the control of the “master” computer 84.

[0020] In the present invention, various fields of view arecontemplated, including a main viewer 90, a left clipboard viewer 92,and a right clipboard viewer 94. As described below, the main viewer 90will display an image which is then being studied, and the left andright clipboard viewers 92 and 94 serve auxiliary display functions.

[0021] In an embodiment of the interpretation unit 22 in which threepersonal computers are networked together, the monitor of the mastercomputer 84 acts as the main viewer 90, the monitor of the left slavecomputer 86 will act as a left clipboard viewer 92 and the monitor ofthe right slave computer 88 will act as the right clipboard viewer 94.However, the viewers 90, 92 and 94 may also be implemented as separatewindows displayed on a single monitor if interpretation unit 22comprises a single personal computer or workstation. In addition, sincepatient medical images at this time generally consist of black and whiteimages, it would also be possible to implement the three viewers asseparate monitors driven by a single personal computer or workstation byappropriate software to drive each monitor in one of the R,G,B (red,green, blue) channels of the RGB color model.

[0022] In a preferred embodiment, the three monitors of interpretationunit 22 are mounted on a pivoting mountings, as shown in FIGS. 6 and 7.The pivoting mountings position the main viewer monitor 90 in a centralposition, with the left and right clipboard viewer monitors 92 and 94angled inwardly to provide a convenient viewing of all three monitorssimultaneously, as shown in FIG. 6. The pivoting mountings are alsodesigned so that the left and right clipboard viewer monitors may bepivoted outwardly, to allow for use of the left and right clipboardviewer monitors 92 and 94 as separate main viewers by other medicalpersonnel, as shown in FIG. 7.

[0023] Interpretation unit 22 will have installed in it a “Viewer”application software running on Windows 95 or Windows NT that respondsto the diagnosing physician's use of the controllers 20, 120 toimplement the commands transmitted from controllers 20, 120. Whereinterpretation unit 22 consists of three separate computers, eachcomputer will have the “Viewer” software loaded on it, allowing eachcomputer and its associated monitor to be simultaneously used bydifferent persons if desired. The use of separate computers with“Viewer” software provides backup systems in the event of the failure ofthe “master” computer 84.

[0024] In the expected environment of use, the interpretation unit 22will have received a digital “folder” for the patient to be diagnosed.The “folder” will contain patient identifying information, a patientmedical history and/or chart; and a series of digital images to bereviewed by the diagnosing physician. Unlike current filmless systems,the patient images stored in the folder will be images selected by askilled technician in the same way as is now done in conventionalradiology film systems. The images in the patient file will be capturedat the MRI or other imaging unit by placing a suitable digitizing tabletin the location where films are exposed by the technician. The exactsame procedures as in selecting images for film exposure will apply tothe filmless images to be captured for use in the present invention. Thedigital images are preferably provided in a bitmap format, althoughother formats may be used.

[0025] The patient “folder” will typically be named with the name of thepatient, e.g. “John Smith.” The interpretation unit 22 may receive suchpatient information via a hospital network, or a modem transmission, ordirectly from an image capture device such as a MRI unit.

[0026] Controller 20 uses a pistol grip with a control end 26 located inan upper end, similar in appearance to a computer joystick used forplaying computer games. Controller 120 is a more elongate handhelddevice with a control end 26 at one end. In the specification followingthe term “controller 20” shall be used to refer to both controllers 20and 120, as the functions thereof are identical, the sole differencebeing in their shape.

[0027] Controller 20 is preferably shaped to be comfortably held in onehand. At control end 26 of case 24 are provided the controls foroperating the image control subsystem and the dictation controlsubsystem.

[0028] In the preferred embodiment, the controller 20 has only fourswitch elements or assemblies to operate as the dictation control andthe image display control: a trigger switch 42 located at the underside28 of controller 20; a pressure switch 44 located on the upper surface29 of controller 20; and a left toss button 46 located on the uppersurface 29 of controller 20 to the left of pressure switch 44; and aright toss button 48 located on the upper surface 29 of controller 20 tothe right of pressure switch 44. With reference to FIG. 1, the triggerswitch 42 is located where it may be actuated by the forefinger of theuser, while the pressure switch 44 and the left and right toss buttonsare located on the upper surface of controller 20 where they may beactuated by the user's thumb. These four switch elements or assembliesprovide all necessary inputs to achieve the navigation of images anddictation control. If desired, of course, additional buttons andcontrols may be added, but it is expected that a minimum of controlswill provide the most reliable and user friendly control. The presentinvention desirably has no more than six switch elements or assembliesfor operation of all necessary functions. The reduction of controls isachieved in the present invention by allowing for different types ofinputs to each switch assembly, such as double clicking or pressuresensitive switches, that allow for different outputs for each input.This is also achieved in the present invention by providing for thedifferent switch assemblies to operate different functions depending onthe current mode of operation of the image display and the audiodictation, or combination thereof, at the time of the input to thecontrol.

[0029] It is to be appreciated that other switch elements may besubstituted for the preferred switch embodiments described above inconnection with the controller 20. For example, in an alternativeembodiment, the functions of pressure switch 44 and left and tight tossbuttons 46 and 48 described below may be implemented in a joystick typecontrol that is movable from front to back and left to right, or by afour way rocker switch, or a thumbwheel, or a tiny trackball, or asensitive touchpad, or set of directional arrow keys located in thecontroller 20. Such variations are all within the scope of theinvention.

[0030] In addition to the control switches 42, 44, 46, and 48,controller 20 is provided with a microphone 50 to receive voicedictation and convert the dictation to appropriate electrical impulses.In another embodiment, controller 20 may have a jack 52 for receiving aclip-on microphone instead of, or in addition to, the microphone in thecontroller 20. Microphone 50 is preferably located on the upper surface29 of controller 20 at its control end 26.

[0031] Controller 20 may be attached to interpretation unit by suitableelectrical cabling, however, in the preferred embodiment, controller 20is a wireless unit that remotely communicates with interpretation unit22. Such wireless systems are known in the art and may include radiofrequency, infrared, or other wireless communications systems.Preferably, controller 20 is provided with a infrared transmitter 52 atits control end 26 for communication with the interpretation unit 22. Insuch case, the various keyboard interrupts and/or mouse interruptsgenerated by the controller 20 are communicated via modulated infraredbeams transmitted from the infrared transmitter 52 to infrared receivers54 associated with the computers or workstations of interpretation unit22 to cause the desired actions. In addition, voice dictation receivedin microphone 50 and converted to electrical signals are alsocommunicated from the infrared transmitter 52. If desired, a singletransmitter 52 may serve to communicate both the control signals and thedictation signals to the infrared receiver, or separate infraredtransmitted may be used. In a wireless embodiment, controller 20 isprovided with a power source such as suitable batteries to power thefunctions of the controller 20.

[0032] The control switches 42, 44, 46 and 48 generate keyboardinterrupt and mouse control signals that control the interpretation unit22. Such keyboard interrupt and mouse control signals are conventionalsignals in operation of personal computer operating systems which aregenerated, respectively, upon the entry of particular keystrokes orclicking the mouse buttons. In a conventional system, for example aMicrosoft Windows operating system, event routines are actuated uponpressing a keyboard key as follows: the keyboard chip transmits amessage, that a key or combination of keys are pressed, to a buffer,which notes the key and transmits a message identifying an “onkeydown”routine and the key or key combination pressed to the operating system.The operating systems receives this message and posts a message to allopen programs when a keyboard interrupt is received; and the programwhich is the active program (known as the program which has the “focus”)responds to the keyboard input. Upon release of the key on the keyboard,the key chip transmits another key indication; the buffer notes the keyand transmits a message identifying an “onkeyup” and the key or keycombination pressed to the operating system, which then again transmitsthis message to all open programs, and the active program respondsthereto. In the present invention, the controller switches actuate suchkeyboard interrupts to instruct a “Viewer” application software runningon one or more of the master and slave computers 84, 86, and 88.

[0033] It is to be appreciated that with the present invention, thecontroller 20 is intended to be the only necessary control, and thus theconventional keyboard and mouse controls may be omitted from thepersonal computers that constitute the interpretation unit 22. However,of course, if desired, such interface devices may be included in theinterpretation unit 22, as for example, if the present invention isimplemented in a single computer at a medical office where the computermay be used for other functions besides review of medical images fordiagnosis. However, it is expected that when the “Viewer” software foroperation of the personal computers in their function as aninterpretation unit 22 is running, that the controller 20 will be theonly interface device necessary for a diagnosing physician's control ofthe interpretation unit 22 to review patient images and to dictate amedical study.

[0034] In one preferred embodiment, the following switches andcombinations of switches actuate the following functions. However, ofcourse, the various functions described below may be assigned to otherswitches or combinations of switches than described in connection withthe preferred embodiment, such different assignments being all withinthe scope of the invention. As described below, image display controland the dictation control have multiple functions, and at least some ofthe functions of the image display control and the dictation control areboth accomplished through commands generated by actuation of one or moreof the switches. The particular control function provided by a switch orcombination of switches is dependent on the mode of dictation or mode ofimage display or combination of modes of dictation and image display atthe time of operation of such one or more switches.

[0035] The trigger switch 42 actuates the primary functions of the voicedictation system. If the trigger switch 42 is clicked and held down, themicrophone 50 is connected to a power source, and a keyboard interruptis sent to the interpretation unit 22, where the “Viewer” softwaresummons the “audio record” function, and voice dictation by thephysician into the microphone is communicated to the interpretationstation 22 and is recorded as a digital file.

[0036] Recording continues until the trigger switch 42 is released,which is transmitted as a keyboard interrupt that causes the “Viewer”software to summon a “stop audio record” function. A hold down of thetrigger switch 42 will initiate the record function whether in stop orplayback mode.

[0037] The audio record function, as well as rewind, playback and otheraudio functions described herein are conventional features in theWindows 95 operating system and are not described in detail herein. Itis noted that a sound card will be required to cause the analog todigital conversion needed to store the voice dictation as a digitalfile. Voice dictation received by the interpretation unit 22 is storedas a Windows “.wav” file for playback using the conventional Windowsbased audio software. The file may be stored locally on theinterpretation unit 22 or in a network file server as selected.

[0038] The “Viewer” software will automatically save the voice dictationfile each time the trigger switch 42 is released. The “Viewer” softwarewill automatically save voice dictation file under a file name such as“Study Dictation.wav.” The file will preferably be automatically savedin the patient “folder.”

[0039] When the audio system is in neither the record nor playback, ifthe trigger switch 42 is clicked and released, the playback function isactivated. In the audio file currently open by the “Viewer” software,the recorded audio file will back up two seconds from the end of thefile and then will playback that portion of the file to the end. Whenthe recorded audio file is being played back, actuation of the triggerswitch 42 by a click and release stops the playback. Also, when therecorded audio file is being played back, actuation of the triggerswitch 42 by two clicks and release causes the “Viewer” software to jumpto the end of the recorded audio file.

[0040] When the audio system is in neither the record nor playback, ifthe trigger switch 42 is clicked twice it will initiate a “rewind”sequence, in which the position in the audio file is moved back, withacceleration after an initial period of “rewind”. “Rewind” may be haltedby clicking trigger switch 42 once.

[0041] When the audio system is in playback mode, as a result of asingle click of trigger switch 42, toss buttons 44 and 46 can be used tocall additional audio dictation functions. For example, the left tossbutton 44 can be used to toggle through various dictation functions(with a screen display in a small window showing the various functionsselected). These functions preferably include (1) insert additionalaudio; (2) overwrite audio; and (3) delete audio. Once a dictationfunction is selected, the right toss button 46 may be clicked during thedesired position in playback to mark the position of the desiredfunction, and clicked a second time, where applicable, to mark the endof the function. Thus, where insert additional audio is desired, theright toss button 46 will mark the position of the audio insert; and thetrigger switch 42 may be activated to record the audio insert. Whereoverwrite audio is desired, the right toss button 46 may be clicked tomark the beginning of the portion of the audio file to be overwritten,and clicked again to mark the end of the portion to be overwritten, andthe trigger switch 42 may be activated to record the audio insert to theoverwritten portion. Where delete audio is desired, the right tossbutton 46 may be clicked to mark the beginning of the portion of theaudio file to be deleted, and clicked again to mark the end of theportion to be deleted.

[0042] Other dictation switch designs may also be used in the handheldcontroller 20. For example, the dictation control switch may be a slideswitch actuated by forward and rearward motion. In such an embodiment,the standard dictation controls (record, off, play, fast forward,rewind) are actuated by positioning the slide switch along thelongitudinal axis of the controller 20. Thus, recording is actuated bypositioning the slide switch at the forward position of its travel.Moving the slide switch backwardly actuates the following modes, inorder: stop, playback, fast forward, and rewind. Other switch controlsystems may also be implemented in accordance with the invention.

[0043] The audio file comprising the diagnosing physician's dictatedstudy of the patient images may be converted to a text file by atranscriptionist or by use of voice recognition software. At the time ofthe present invention, it is expected that a transcriptionist will berequired to accurately transcribe the dictated study. However, asvendors such as IBM and others continue to improve voice recognitionsoftware, that the services of a transcriptionist may be less needed,and that voice recognition software may be used to convert the recordedaudio file to a text file.

[0044] In a system using a transcriptionist in a distributed network, acentral file server will maintain information on the status of thepatient folder and the contents thereof, and will forward a copy of theaudio to a transcription station to generate the corresponding textfile.

[0045] The controller 20 also controls the “Viewer” software to permitthe physician to review the images of the patient. Pressure switch 44 isthe principal control for the image viewing system. Pressure switch 44is a strain gauge switch that transmits different keyboard keys askeyboard interrupt instructions to the interpretation unit 22 dependingon the amount of pressure applied to the switch. In the preferredembodiment, pressure switch 44 is a thumb actuated rocker bar switchthat has a default centered position, and which may be pushed forwardlyor pulled backwardly, against a bias that will tend to center thepressure switch in its centered position. Forward pushing on the edge 45of switch 44 will cause the Viewer software to advance through the imageseries. Backwards pulling thereon will cause the Viewer software to backup through the image series. The different keys transmitted by pressureswitch 44 instruct the interpretation station 22 to advance or back upthrough and display the image series at different speeds.

[0046] The image series stored in the patient folder in interpretationunit 22 is preferably stored as multiple files, identified for exampleas 100010001, 100010002, where the terminal four digits identify thenumber of the image in the series, and the second through fifth digitsindicate the number of the series. Actuation of the pressure switch 44by a single push will cause the Viewer software to display each imagefile in numeric order, one image at a time, beginning from the firstimage of the first series. Actuation of the pressure switch 44 bypushing and holding the pressure switch 44 will cause the Viewersoftware to scroll through the images. The amount of pressure applied topressure switch 44 will control the speed at which the Viewer softwarescrolls through the images in the series. The scrolling of images willhalt when the end of an image series is reached (or the end of the setof all images reviewed during the study so far), whereupon the firstimage of the next series will be displayed, and the scrolling functionwill be terminated. The user may then step or scroll through thedisplayed next image series by again activating the pressure switch 44.In the preferred embodiment, the physician will be required to stepthrough the entire sets of image series before rapid scrolling will bepermitted. This will help to insure that each image is reviewedinitially, to prevent missing a potentially relevant image during rapidscrolling.

[0047] A serial display as described above is believed to be the mosteffective way of presenting the images in the patient file, as itmaximizes the diagnosing physician's ability to interpret the imageusing lower cost, standard size computer monitors. However, it is alsopossible to present the image information in a matrix format withmultiple images displayed in a monitor at one time. Such multiple imagesmay comprise a tiled set of windows, each containing a separate image.In such case, the controls of controller 20 will be as described above,except that in stepping or scrolling through images, the Viewer programwill step through the different windows.

[0048] However, it is believed that a serial display provides apreferable approach as it will force the physician (particularly theexperienced physician) to review all the images presented, without thepossibility of skipping over a potentially relevant image. With anexperienced physician there is sometimes a temptation to leap to themost relevant image—and to reach the obvious diagnosis. With a serialdisplay, a physician will be less likely to miss the less obviousdiagnosis that might be reached after proper review of all images. Theserial image display is also helpful to the inexperienced physician whowill have a better awareness of the entire image set, and what he islooking at, by virtue of the preceding images already displayed. A finalbenefit of a serial display approach (as opposed to a multiwindowformat) is that it allows presentation of a large image. In comparisonto film, which has a quality of very high contrast resolution at acertain luminance, current monitors provide much less resolution. Tocompensate for the reduced contrast resolution, the spacial resolutionis enhanced (the image is made larger) to give the user an image ofsubjective quality that approaches the resolution provided by film.

[0049] The “Viewer” software will also determine if the user hasdisplayed a specific image for a predetermined period of time. If so,the Viewer software will “bookmark” for subsequent return. The“bookmark” may be inserted by adding an appropriate stop in the patientimage file, or in a table containing a list of the images, or by storingthe filename of the displayed image for later recall. Thus, if thediagnosing physician studies the third image in a first series for aperiod exceeding the predetermined period of time, then steps forward tothe third image of the next series, the physician may jump back to thethird image of the first series by pulling back on the pressure switch44. The “Viewer” software, having bookmarked the third image of thefirst series, will stop backwards scrolling when that image isdisplayed. In another embodiment of the invention, images studied for apredetermined period of time may be selected to be automatically tossedto one of the clipboard viewer monitors; and/or tossed images may beauto-bookmarked.

[0050] As mentioned above, interpretation unit 22 preferably comprisesthree separate networked personal computers or workstations, operating amain viewer monitor 90, a left clipboard viewer monitor 92, and a rightclipboard viewer monitor 94. In a default mode of operation, the mainviewer monitor 90 will have displayed on it the selected displayed imagein an image series. The left clipboard viewer 92 will be set up todisplay a number of images in a tiled format, for example, in a 2×2matrix of tiled images. Other matrix sizes may be used, however, forbetter image clarity, a 2×2 matrix size is preferred. A title block willbe associated with each displayed image to identify the filename of theimage.

[0051] The toss buttons 46 and 48 serve two functions. A quick click ofa toss button 46 or 48 causes the image currently on view on the mainviewer monitor 90 to be separately displayed on the respective clipboardviewer monitor 92 or 94. This is again accomplished by the toss buttongenerating a keyboard interrupt with instructs the “Viewer” program todisplay the selected image on the selected clipboard viewer monitor, forexample, by copying the image to a new file containing images ofinterest, and displaying the new file. The use of the toss buttons 46and 48 in this manner also marks the “tossed” image for later use of thecomparison function described below. An image that has been “tossed” toa clipboard viewer monitor may be “untossed” (removed from the clipboardviewer) by clicking the toss button for the clipboard viewer when themain viewer monitor 90 has displayed on it the image to be “untossed”.

[0052] It is anticipated that the right clipboard viewer will generallybe used for assembling a summary study containing images of particularmedical interest, while the left clipboard viewer will be used as aworksheet for displaying side by side comparisons of different series,or display of prior patient studies for comparison (or consultation asneeded without interruption of the present study).

[0053] If a toss button 46 or 48 is pressed and held down, it willinvoke a “mode selection” function in the main viewer program. The“Viewer” program will display a menu of the various modes on the mainviewer monitor or one of the clipboard monitors. In the “mode selection”mode, the pressure switch 44 may be used to cycle through the variousmodes, accompanied by a display in which a highlighted selection bardown the menu of modes. A mode may then be selected by clicking on thetoss button again. Of course, other control schemes may be implementedusing the controller 20, for example, upon entering a “mode selection”mode, the toss buttons could be used to cycle through the various modesby clicking the toss buttons to cycle one entry at a time through thevarious modes, and selection could be made by clicking the triggerswitch 42; or the trigger switch 42 could be used to cycle through thevarious modes and the selection could be made by pressing one of thetoss buttons. If the toss buttons are used to cycle through the variousmodes, one toss button will cycle up through the mode selection menu andthe other toss button will cycle down through the mode selection menu.In another embodiment, one toss button might be used to cycle throughthe mode selection menu and the other toss button would be used toselect the particular mode to be implemented. Other mode selection andselection schemes are equally within the scope of the present invention.

[0054] The “modes” to be selected from may include one or more of thefollowing functions.

[0055] (1) List Patients mode. This will cause the system to display thepatient folders so that the folder of a patient to be studied can beselected.

[0056] (2) View mode. This will cause the system to return to thedefault viewing mode with controller 20 controlling the master computer84 of the interpretation unit 22.

[0057] (3) Compare mode. This will cause one clipboard viewer(preferably the left clipboard viewer monitor 92) to display aside-by-side presentation of all images in two series starting with thefirst marked image in each series. For example, while in view mode, adiagnosing physician could select the third image of the second seriesand send it to the left clipboard viewer 92 by marking with toss button46, and could then select a fourth image of the fourth series and sendthis image to the left clipboard viewer 92 by marking with toss button46. The images would then be displayed with the third image in thesecond series in the top left quadrant of the monitor screen, and thefourth image of the fourth series in the top right quadrant of themonitor screen. The diagnosing physician would then summon the comparemode, which would cause the display of all subsequent images to theselected images in a side by side relationship, with the selected imageson top, and the subsequent images below.

[0058] (4) Annotate mode. The master computer 84 may be set up with alight pen, allowing the diagnosing physician to annotate the patientimage displayed on the main viewer 90. The annotated image may then besaved as a separate file from the original image by using toss button 48to save it to the right clipboard viewer 94, or upon return to viewmode. Screen buttons actuated by the light pen may be provided to stepforward or back through the image series and to “toss” an annotatedimage to the left or right clipboard. A submenu may also be establishedto select the particular image to be annotated.

[0059] (5) Left clipboard control. This will cause the controller 20 tobe acting on and controlling the left clipboard viewer monitor 92. Thusfor example, after invoking the compare function described above, thediagnosing physician could then select left clipboard control and usethe controller 20 to step up and down or scroll up and down through theside by side comparison, using the pressure switch 44 and the navigationcontrol.

[0060] (6) Right clipboard control. This will cause the controller 20 tobe acting on and controlling the right clipboard viewer monitor 94.

[0061] (7) Dictation special functions. This menu selection will call asubmenu with the insert, overwrite, and delete the dictated audio filefunctions described above.

[0062] (8) End study. This will cause the system to close the open filesbeing viewed, and the dictated audio file, and to save the summary studyprepared by the diagnosing physician as a separate file or files.

[0063] Other modes might include a main menu mode, to call a main modeselection menu, if, for example, some or all of the above functions oradditional functions are established as submenus to be selected; and aMark image mode to bookmark the image currently on display in the mainviewer, or to bookmark other images as may be a selected from a submenu,to subsequently implement comparison and other functions. The light penmight also be used by a non-physician technician to operate variousother functions of the computer.

[0064] When series of images in a patient study is first a opened, theclipboards may be used to provide an overview of the study. For example,the left and right clipboard viewers 92 and 94 might first havedisplayed in them the center image of each series of images. Other modesof initial display may be provided, such as the first and last image ofa series, or all of the images in one or more series in the patientimage file. In some patient image studies, there is a “scout slice”,which is an orthogonal view of the patient anatomy of interest showingthe location of the various slices constituting the image series. Wherea scout slice is available, it will preferably remain on display, mostpreferably as the last image on display on the left clipboard viewermonitor 92.

[0065] Upon tossing of a first image to a clipboard viewer monitor bythe toss function, the initial image display will be cleared. Thus forexample, upon invoking the toss function to transfer a copy of the imageon the main viewer monitor 90 to the right clipboard viewer monitor 94,the images placed on the right clipboard viewer monitor 94 at initialstart-up will be cleared. As noted, it is anticipated that the rightclipboard viewer 94 will generally be used for assembling a summarystudy containing images of particular medical interest, and may includeannotated images as described above, while the left clipboard viewer 92will be used as a worksheet. When the summary study is completed, itwill be saved as a separate file from the original patient study. It isalso possible to save the summary study in the form of a list specifyingthe images of particular interest from the original, complete, patientimage file. The summary study images will typically be the imagespertinent to the medical diagnosis, and may printed, and/or deliveredback to a referring physician along with the text file comprising thedictated patient study, either in electronic or hard copy form.

[0066] The clipboard viewer monitors may also be used for a comparativeimage display as described above.

[0067] The clipboard viewers 92 and 94 also serve other functions. Asnoted above with regard to the master computer, one slave computer 86 or88 may be set up with a light pen, allowing the diagnosing physician toannotate the patient image displayed on the clipboard viewer. In suchcase, the patient image will most likely be displayed in a full sizeimage, rather than in a tiled matrix of smaller images, although, ofcourse, such a smaller format may also be used. Also, the clipboardviewers 92 and 94 could be used to pull up and review a prior patientstudy without closing the study in progress, as, for example, if thediagnosing physician receives a telephone call seeking discussion of apast patient study, the diagnosing physician could call up, review, anddiscuss the past patient study, and then return the clipboard viewer toits slave functions as described above.

[0068] Control of the display on the clipboard viewer monitors may beaccomplished via controller 20 or by separate controls associated witheach of the main and clipboard viewers. When control is transferred to aslave computer, another copy of the “Viewer” software is launched in theslave computer and all functions thereof are available on the slavecomputer.

[0069] The display on the clipboard viewer monitors, as well as on themain viewer monitor, may also be controlled by means of an array ofbuttons placed on or near the monitors. Such buttons, like controller20, generate keyboard interrupts that provide the control of the imageson the viewer monitor. For example, buttons may be provided that triggera stepping forwards or backwards, or scrolling forwards or backwards,through the images that are selected for display on the particularmonitor. Of course, other controls implemented through other interfacessuch as a conventional keyboard, mouse, touch screen, light peninterface, and the like may also be used.

[0070] As prior noted the various functions described above may beassigned to a switch or combination of switches other than the describedswitch or combination of switches. In this respect, the invention isdesigned to provide a “context sensitive” control in which variousswitch combinations achieve different results. For example, as describedabove, in a dictation edit/revision mode, the physician will not belikely to need the “toss” or “compare” functions, so that the tossbuttons 46 and 48 (or other switch elements) may be assigned dictationcontrol functions instead of their usual toss functions. It would alsobe within the scope of the invention to disable the image advance andretreat functions of a switch such as pressure switch 44 during adictation edit mode and to assign dictation control functions to thatswitch during dictation edit mode. Similarly, as described above, whenthe “mode selection” menu is displayed, the physician is not likely toneed the dictation functions, and the trigger switch 42 may be assignedmode selection functions instead of the usual dictation functions.

[0071] A further feature of the present invention is a “multimedia”record function. This function will desirably be implementedautomatically whenever the dictation trigger switch is activated, or maybe limited to the final “Conclusion” which is conventional at the end ofany medical diagnostic dictation, or may be applied in some othersummary time period. The “multimedia record function” is to record theimages being reviewed by the physician at the time the dictation isrecorded, enabling a later playback of the dictation with simultaneousdisplay of the images that the physician reviewed at the time of thedictation. The image display will be synchronized to the dictation sothat on playback the precise image being viewed at the time of thespecific dictation is displayed. The “multimedia record function” may beimplemented by recording the file name of the image on display at thetime of particular dictation, which may be embedded into the dictationfile as a link or using other conventional techniques for slide showtype presentations, or by other file list methods.

[0072] The above described serial display system for patient images isbelieved to be the most efficient and user friendly system that may beimplemented. However, other possible control systems may also beimplemented in accordance with the invention. In an alternative controlsystem, using a pressure switch having additional left and right switchpositions in addition to front and back switch settings, it would bepossible to control the “Viewer” software to jump among the differentimage series. Thus instead of scrolling to the end of an image series toobtain the next series, a diagnosing physician could implement thedisplay of the next preceding series at the same image number as beingcurrently viewed by leftward pressure on the switch, or of the nextsucceeding series by rightward pressure on the switch. This would permit“lateral” movement among the various patient series. Once a new seriessequence was selected, the physician could then scroll forwards orbackwards through the series as in the above described serial displayembodiment. An adjunct clipboard viewer display might also be providedin such case to provide a matrix containing thumbnail size images of allthe images in the patient folder, arrayed in columns, with each columncontaining the images of a different series. The adjunct clipboardviewer would indicate the image being presented to the main viewer byappropriate highlighting or framing, as an aid to navigation among theimages in the study.

[0073] Although the foregoing description has described an applicationof a combined image review/voice dictation function in a controller 20operating in conjunction with a medical interpretation unit, the presentinvention has broader application as well. It may be implemented forexample in a legal or business environment. For example, usingcontroller 20, an attorney reviewing documents could dictate comments onthe documents (as in reviewing electronically stored raster images ofproduction documents in litigation) or revisions to a document (as inrevisions to a contract). The images which may be controlled in thisfashion may thus include graphic images, text documents, and otherelectronic files. The controller 20 in the first example would interfacewith a viewer software that controls viewing of the imaged documents.The controller 20 in the second example would interface with wordprocessing software to insert a comment or margin note in the documentfor a transcriptionist to later type in the requested changes, or wouldwork in conjunction with a speech to text conversion system to implementthe revisions. Similarly, a business person might dictate comments orresponses to electronically received faxes, and documents for typing orconversion to text.

[0074] The present invention provides an convenient and practicalapparatus for review of radiographic medical images with integrateddictation control and has better operational flexibility than any deviceknown in the art.

[0075] It is to be appreciated that the foregoing is illustrative andnot limiting of the invention, and that various changes andmodifications to the preferred embodiments described above will beapparent to those skilled in the art. For example, the invention hasapplication in other fields where it is desired to have a handheld unitthat can control image review with dictation abilities, such as in legaland business fields. Such changes and modifications can be made withoutdeparting from the spirit and scope of the present invention, and it istherefore intended that such changes and modifications be covered by thefollowing claims.

What is claimed is:
 1. Apparatus for annotation and navigation ofelectronic images for use with a computer system, said computer systembeing provided with at least one image display unit, random accessmemory, means for digital data storage, comprising: a handheldcontroller, said controller having a finger operated image displaycontrol for generating commands for controlling display of images insaid image display unit, a finger operated dictation control forgenerating commands for controlling recording and playback of audiodictation, and means for communicating said commands for controllingdisplay of images, said commands for controlling recording and playbackof audio dictation, to said computer system to control said imagedisplay and to a means for recording and playback of said audiodictation.
 2. Apparatus in accordance with claim 1, wherein said meansfor recording and playback of said audio dictation comprises digitaldata storage provided in said computer system operable by an operatingsystem in said computer using digital to analog circuitry.
 3. Apparatusin accordance with claim 2, further comprising a microphone forreceiving audio dictation and generating electrical signalscorresponding to said audio dictation.
 4. Apparatus in accordance withclaim 3, wherein said means for communicating further communicates saidelectrical signals corresponding to said audio dictation to saidcomputer system for digital storage therein, and whereby said dictationcontrol is operable to control recording and playback of said digitallystored electrical signals corresponding to said audio dictation. 5.Apparatus in accordance with claims 1, 2, 3, or 4, wherein said imagedisplay control and said dictation control have multiple functions, andwherein at least some of said functions of said image display controland said dictation control are both accomplished through commandsgenerated by actuation of one or more switch elements, the particularcontrol function provided by a switch element or combination of switchelements being dependent on a mode of dictation or a mode of imagedisplay or combination of modes of dictation and image display at thetime of operation of said one or more switch elements.
 6. Apparatus inaccordance with claim 1, 2, 3, or 4, wherein said image display controland said dictation control comprise no more than six switch elements. 7.Apparatus in accordance with claim 6, wherein said image display controland said dictation control comprise no more than four switch elements.8. Apparatus in accordance with claim 1, wherein said dictation controlin said controller comprises a dictation switch mechanism located insaid controller, said dictation switch mechanism being operable togenerate said commands for recording and playback of audio dictation. 9.Apparatus in accordance with claim 8, wherein said dictation switchmechanism comprises a trigger switch.
 10. Apparatus in accordance withclaim 1, wherein said image display control comprises: a centered switchmechanism that has neutral centered position, and which is responsive toforward pressure to cause said centered switch mechanism to generate acommand to cause said computer system to advance through a series ofimages displayed in said image display unit, and which is responsive torearward pressure to cause said centered switch mechanism to generate acommand to cause said computer system to back up through a series ofimages displayed in said image display unit.
 11. Apparatus in accordancewith claim 10, wherein said centered switch mechanism is responsive tothe amount of pressure applied thereto whereby lesser pressures appliedthereto generate commands for advancing or backing up through images ata slower rate, and greater pressures applied thereto generate commandsfor advancing or backing up through images at a faster rate. 12.Apparatus in accordance with claim 11, wherein said centered switchmechanism is a strain gauge.
 13. Apparatus in accordance with claim 10,wherein said image display control further comprise left and rightswitch mechanisms to generate a command to cause said computer system toselect a displayed image in said display unit for further study. 14.Apparatus in accordance with claim 13, wherein said left and rightswitch mechanisms further may be operated to generate a command to causesaid computer system to display a menu of available functional modes.15. Apparatus in accordance with claim 13, wherein said left and rightswitch mechanisms further may be operated to generate commands tocontrol audio dictation functions in said computer system.
 16. Apparatusin accordance with claim 4, wherein said controller is a wirelesscontroller provided with wireless communication means for communicatingsaid commands for controlling display of images, said commands forcontrolling recording and playback of audio dictation, and saidelectrical signals corresponding to said audio dictation to saidcomputersystem.
 17. Apparatus in accordance with claim 16, wherein said wirelesscommunication means comprises an infrared transmitter provided in saidcontroller and an infrared receiver associated with said computersystem.
 18. A handheld controller for controlling annotation andnavigation of electronic images, for use with a computer system, saidcomputer system being provided with at least one image display unit,random access memory, means for digital data storage, comprising: fingeroperated image display controls for controlling display of images insaid image display unit, said image display controls including a firstswitch mechanism for generating commands to cause said computer systemto advance through a series of images displayed in said images displayunit and to back up through said series of images displayed in saidimage display unit, and second and third switch mechanisms to generateadditional commands to cause said computer system to perform additionalfunctions to permit further study of a displayed image in said displayunit; a microphone for receiving audio dictation and converting audiosignals to electrical signals; a finger operated dictation controlhaving a dictation switch mechanism located in said controller, saiddictation switch mechanism being operable to generate commands forrecording and playback of audio dictation received in said microphone,converted to said electrical signals, and digitally stored in saidcomputer system; and communication means for communicating commandsgenerated by said image display controls and said dictation controls,and said electrical signals from said microphone, to said computersystem.
 19. A controller in accordance with claim 18, wherein said imagedisplay control and said dictation control have multiple functions, andwherein at least some of said functions of said image display controland said dictation control are both accomplished through commandsgenerated by actuation of one or more of said first, second, third anddictation switch elements, the particular control function provided by aswitch element or combination of switch elements being dependent on amode of dictation or a mode of image display or a combination of modesof dictation and image display at the time of operation of said one ormore switch elements.
 20. A controller in accordance with claim 18,wherein said image display control and said dictation control compriseno more than six switch elements.
 21. A controller in accordance withclaim 20, wherein said image display control and said dictation controlcomprise no more than four switch elements.
 22. A controller inaccordance with claim 18, wherein said first switch mechanism has aneutral centered position, and which is responsive to forward pressureto cause said first switch mechanism to generate a command to cause saidcomputer system to advance through a series of images displayed in saidimage display unit, and which is responsive to rearward pressure tocause said first switch mechanism to generate a command to cause saidcomputer system to back up through a series of images displayed in saidimage display unit.
 23. A controller in accordance with claim 22,wherein said first switch mechanism is responsive to the amount ofpressure applied thereto whereby lesser pressures applied theretogenerate commands for advancing or backing up through images at a slowerrate, and greater pressures applied thereto generate commands foradvancing or backing up through images at a faster rate.
 24. Acontroller in accordance with claim 22, wherein said first switchmechanism is a strain gauge.
 25. A controller in accordance with claim22, wherein said dictation switch mechanism comprises a trigger switchlocated for actuation with a forefinger of a user of said controller,and said first switch is located on an upper surface of said controllerfor actuation with a thumb of the user of said controller.
 26. Acontroller in accordance with claim 18, wherein said communication meanscomprises an infrared transmitter provided in said controller and aninfrared receiver associated with said computer system.
 27. A controllerin accordance with claim 18, wherein said second and third switchmechanisms further may be operated to generate a command to cause saidcomputer system to display a menu of available functional modes.
 28. Acontroller in accordance with claim 18, wherein said second and thirdswitch mechanisms further may be operated to generate commands tocontrol audio dictation functions in said computer system.
 29. Ahandheld controller for controlling dictation and navigation ofelectronic medical images, for use with a computer system, said computersystem being provided with at least one image display unit, randomaccess memory, means for digital data storage, comprising: fingeroperated image display controls for controlling serial display of imagesof a patient displayed in said image display unit, said image displaycontrols including a centered switch mechanism that has neutral centeredposition, and which is responsive to forward pressure to cause saidcentered switch mechanism to generate a command to cause said computersystem to advance in serial order through a series of images displayedin said image display unit, and which is responsive to rearward pressureto cause said centered switch mechanism to generate a command to causesaid computer system to back up in serial order through a series ofimages displayed in said image display unit, said centered switchmechanism being responsive to the amount of pressure applied theretowhereby lesser pressures applied thereto generate commands for advancingor backing up through images at a slower rate, and greater pressuresapplied thereto generate commands for advancing or backing up throughimages at a faster rate, and left and right switch mechanisms located onleft and right sides of said centered switch mechanism to generateadditional commands to cause said computer system to perform additionalfunctions to permit further study of a displayed image in said displayunit; a microphone for receiving audio dictation and converting audiosignals to electrical signals; a finger operated dictation controlhaving a first switch mechanism located in said controller, said switchmechanism being operable to generate commands for recording and playbackof audio dictation received in said microphone, converted to saidelectrical signals, and digitally stored in said computer system; andwireless communication means for communicating commands generated bysaid image display controls and said dictation controls, and saidelectrical signals from said microphone, to said computer system.
 30. Acontroller in accordance with claim 29, wherein said wirelesscommunication means comprises an infrared transmitter provided in saidcontroller and an infrared receiver associated with said computersystem.
 31. A controller in accordance with claim 30, wherein said leftand right switch mechanisms further may be operated to generate acommand to cause said computer system to display a menu of availablefunctional modes.
 32. A controller in accordance with claim 30, whereinsaid left and right switch mechanisms further may be operated togenerate commands to control audio dictation functions in said computersystem.
 33. A controller in accordance with claim 29, wherein said imagedisplay control and said dictation control have multiple functions, andwherein at least some of said functions of said image display controland said dictation control are both accomplished through commandsgenerated by actuation of one or more of said first, second, third anddictation switch elements, the particular control function provided by aswitch element or combination of switch elements being dependent on amode of dictation or a mode of image display or combination of modes ofdictation and image display at the time of operation of said one or moreswitch elements.